Papers with systematic approach
How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Code (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Language models (LMs) have exhibited impressive abilities in generating code from natural language requirements. |
| Approach: | They propose to introduce various metrics with inter-code similarity to evaluate the diversity of generated code by comparing model-generated solutions with human-written ones. |
| Outcome: | The proposed method leverages LMs’ capabilities in code understanding and reasoning, resulting in a set of metrics that represent the number of algorithms in model-generated solutions. |
Data Filtering using Cross-Lingual Word Embeddings (2021.naacl-main)
Copied to clipboard
| Challenge: | varying task definitions and data conditions make it difficult to draw a meaningful comparison. |
| Approach: | They propose to use language identification to perform data filtering on MT data based on cross-lingual word embeddings to identify weaknesses in language identification tool. |
| Outcome: | The proposed methods perform well on three real-life, high resource MT tasks while performing weakly within more realistic task conditions. |
SoundMind: RL-Incentivized Logic Reasoning for Audio-Language Models (2025.emnlp-main)
Copied to clipboard
Xingjian Diao, Chunhui Zhang, Keyi Kong, Weiyi Wu, Chiyu Ma, Zhongyu Ouyang, Peijun Qing, Soroush Vosoughi, Jiang Gui
| Challenge: | Recent large language models have demonstrated impressive reasoning abilities, but their extension to the audio modality remains underexplored. |
| Approach: | They propose a rule-based reinforcement learning algorithm to equip LALMs with robust reasoning capabilities. |
| Outcome: | The proposed algorithm improves on the SoundMind benchmark. |
Low-Perplexity LLM-Generated Sequences and Where To Find Them (2025.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly applied across various domains, but the ways they leverage their training data during inference remains only partially understood. |
| Approach: | They propose a systematic approach that analyzes low-perplexity sequences and traces them back to their sources in the training data. |
| Outcome: | The proposed pipeline extracts low-perplexity sequences across diverse topics while avoiding degeneration, then trace them back to their sources in the training data. |
Know Better – A Clickbait Resolving Challenge (2022.lrec-1)
Copied to clipboard
| Challenge: | a clickbait headline or teaser is used to "bait" the reader into clicking a link to an article . clickbaiting is annoying but effective, and can be countered with specialized models . |
| Approach: | They propose to construct approaches that can automatically extract relevant information from clickbait articles . they argue that clickbaiting can probably not be defeated with clickbaitting detection alone . |
| Outcome: | The proposed methods outperform question answering models on clickbait resolving task . the data will be used to give users tools to counter clickbaiting in the future . |
SimSCOOD: Systematic Analysis of Out-of-Distribution Generalization in Fine-tuned Source Code Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large datasets are increasingly available for pre-training source code models, but obtaining representative training data that fully covers the code distribution for specific downstream tasks remains challenging due to the task-specific nature and limited labeling resources. |
| Approach: | They propose a systematic approach that simulates various OOD scenarios along different dimensions of source code data properties and investigates model behavior under different fine-tuning methodologies. |
| Outcome: | The proposed approach simulates various OOD scenarios along different dimensions of source code data properties and exposes multiple failure modes attributed to OOD generalization issues. |
Taxonomy-Guided Zero-Shot Recommendations with LLMs (2025.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to deploy large language models (LLMs) into RecSys have limited prompt length, unstructured item information, and un-constrained generation of recommendations. |
| Approach: | They propose a taxonomy-guided recommendation framework that empowers LLMs with category information in a systematic approach. |
| Outcome: | The proposed framework significantly improves recommendation quality compared to zero-shot approaches. |
Towards Understanding Counseling Conversations: Domain Knowledge and Large Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing language models such as Transformer-based models fail to predict the conversation outcome. |
| Approach: | They propose to integrate human-annotated domain knowledge and LLM-generated features to provide richer context to counseling conversations. |
| Outcome: | The proposed model improves by 15% when combined with human-annotated domain knowledge and LLM-generated features. |
Complexity-Guided Curriculum Learning for Text Graphs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Curriculum learning is a systematic approach to training that refines training progressively and tailors training to task requirements. |
| Approach: | They propose a curriculum learning approach that employs "spaced repetition" and complexity formalisms to guide the training process. |
| Outcome: | The proposed model gains more and uses less data, and the best curricula are equally effective. |
Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Sampling (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) generate reasoning paths before answers, but lack a systematic approach to determine optimal number of samples or select the most faithful rationale. |
| Approach: | They propose a framework that evaluates the quality of reasoning and consistency of answers for each generated sample and uses criteria-based stopping and weighted majority voting to guide early stopping decisions and rationale selection. |
| Outcome: | The proposed framework outperforms existing methods while maintaining accuracy. |
AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling (2025.findings-acl)
Copied to clipboard
| Challenge: | In this paper, we introduce a suite of math models that excel in solving complex math problems. |
| Approach: | They propose a supervised fine-tuning process that achieves competitive performance across general domains, followed by targeted fine- tuning for the math domain using a carefully curated set of prompts and synthetically generated responses. |
| Outcome: | The proposed model outperforms Qwen2.5-Math-72B-Instruct, GPT-4o and Claude-3.5 Sonnet in the math domain. |
Principled Understanding of Generalization for Generative Transformer Models in Arithmetic Reasoning Tasks (2025.acl-long)
Copied to clipboard
| Challenge: | Existing models excel in arithmetic reasoning but their generalization capabilities are incompletely understood. |
| Approach: | They propose a theoretical framework for understanding the generalization behaviors of transformers in arithmetic tasks, focusing on length generalization. |
| Outcome: | The proposed framework can predict generalization behaviors in transformers with a high translation invariance and base mismatch in modular operations. |
Word Rotator’s Distance (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to measure textual similarity are inconsistent with the word alignment and are empirically inferior to the simple cosine similarity between general-purpose sentence vectors. |
| Approach: | They propose to decouple word vectors into their norm and direction and then grow the norm and directions of word vector. |
| Outcome: | The proposed methods outperform alignment-based approaches on several benchmarks and strong baselines on the semantic textual similarity task. |
Quantification of Large Language Model Distillation (2025.acl-long)
Copied to clipboard
Sunbowen Lee, Junting Zhou, Chang Ao, Kaige Li, Xeron Du, Sirui He, Haihong Wu, Tianci Liu, Jiaheng Liu, Hamid Alinejad-Rokny, Min Yang, Yitao Liang, Zhoufutu Wen, Shiwen Ni
| Challenge: | Existing studies have revealed the robustness degra-dation caused by data distillation. |
| Approach: | They propose a framework to evaluate and quantify model distillation . they aim to identify identity cognition contradictions and analyse multi-granularity response similarities across models to measure the extent of homogenization. |
| Outcome: | The proposed framework addresses two key aspects: (1) Identifying identity cognition contradictions to assess discrepancies in how models perceive and represent identity-related information; (2) Analyzing multi-granularity response similarities across models to measure the extent of homogenization. |
Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysis (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on the robustness of LLMs with few-shot prompting techniques are limited. |
| Approach: | They propose to test the robustness of LLMs in multi-hop reasoning tasks via domain-agnostic perturbations. |
| Outcome: | The proposed model is more sensitive to certain perturbations such as replacing words with synonyms and more robust to few-shot prompting methods. |
DFPE: A Diverse Fingerprint Ensemble for Enhancing LLM Performance (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit inconsistent performance across diverse domains. |
| Approach: | They propose a method that systematically constructs subject-adaptive ensembles by balancing model diversity and competence. |
| Outcome: | The proposed method achieves 17.1% gain over the best single model, reaching 71.4% accuracy on the MMLU-pro benchmark. |
“Who said it, and Why?” Provenance for Natural Language Claims (2020.acl-main)
Copied to clipboard
| Challenge: | generating and publishing content is so easy, we are bombarded with information and are exposed to all kinds of claims. |
| Approach: | They propose a formal definition of provenance graph for a given natural language claim . they evaluate the approach using two benchmark datasets to capture provenance . |
| Outcome: | The proposed method shows initial success in capturing provenance and its effectiveness on claim verification. |
Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models (2025.emnlp-main)
Copied to clipboard
Mehdi Ali, Manuel Brack, Max Lübbering, Elias Wendt, Abbas Goher Khan, Richard Rutmann, Alex Jude, Maurice Kraus, Alexander Arno Weber, Felix Stollenwerk, David Kaczér, Florian Mai, Lucie Flek, Rafet Sifa, Nicolas Flores-Herr, Joachim Koehler, Patrick Schramowski, Michael Fromm, Kristian Kersting
| Challenge: | Existing open-source multilingual datasets rely on heuristic filtering methods restricting both their cross-lingual transferability and scalability. |
| Approach: | They propose a systematic approach that curates diverse and high-quality multilingual data at scale while significantly reducing computational demands. |
| Outcome: | Evaluated empirically across 35 languages, the proposed approach outperforms current heuristic filtering methods like Fineweb2 and improves model training quality and retention rates. |
News2vec: News Network Embedding with Subnode Information (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to embed news as vectors do not integrate features and inter-textual knowledge of news. |
| Approach: | They propose a model that integrates news features and inter-textual knowledge into a dense vector representation. |
| Outcome: | The proposed model can be used to represent news as a dense vector . it is compared with existing models on stock movement prediction and news recommendation tasks . |
A Systematic Approach to Derive a Refined Speech Corpus for Sinhala (2022.lrec-1)
Copied to clipboard
| Challenge: | Despite being large and generic, some languages such as Sinhala are left to underutilize the technology due to the lack of adequate resources. |
| Approach: | They propose to derive a corpus from a publicly available corpus for Sinhala speech recognition using crowdsourcing and web scraping techniques. |
| Outcome: | The proposed corpus reduces the Word-Error-Rate by 15.9%. |
What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning (2025.coling-main)
Copied to clipboard
Yifan Du, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, Jinpeng Wang, Chuyuan Wang, Mingchen Cai, Ruihua Song, Ji-Rong Wen
| Challenge: | Experimental results show that visual instruction tuning improves performance of Multi-modal Large Language Models (MLLMs) to extend the application scope of Large Language Modells, a surge of work augments LLMs with vision encoders to endow the ability of multi-modal cognition and reasoning. |
| Approach: | They propose a systematic approach to create high-quality visual reasoning instructions using a synthesize-complicate-reformulate paradigm. |
| Outcome: | The proposed method improves performance of MLLMs by 27.86% and 27.60% on MME-Perception and MME Cognition. |
IR2: Information Regularization for Information Retrieval (2024.lrec-main)
Copied to clipboard
| Challenge: | Effective information retrieval (IR) in settings with limited training data remains a challenging task. |
| Approach: | They propose a technique for reducing overfitting during synthetic data generation . they use DORIS-MAE, ArguAna, and WhatsThatBook as examples . |
| Outcome: | The proposed technique outperforms previous methods and reduces cost by 50% on three recent IR tasks characterized by complex queries. |
ML-Promise: A Multilingual Dataset for Corporate Promise Verification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Promises shape perceptions and drive decisions, but verification of their fulfillment is difficult due to complexity and volume of commitments . authors propose a new approach to verifying promises in environmental, social, and governance reports . complexity of promises, complexity of evidence, difficulty in verifying their fulfillment a pressing need for new approaches . |
| Approach: | They propose a multilingual dataset that includes English, French, Chinese, Japanese, and Korean . they propose ML-Promise to facilitate in-depth verification of corporate promises . |
| Outcome: | The proposed approach includes promise identification, evidence assessment, and evaluation of timing for verification in multiple languages. |
From Insight to Exploit: Leveraging LLM Collaboration for Adaptive Adversarial Text Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | LLMs can provide substantial zero-shot performance on diverse tasks, but it is crucial to assess their robustness against adversarial inputs. |
| Approach: | They introduce Static Deceptor and Dynamic Deceptr to generate adversarial examples . they produce subtle and natural-looking adversarials that preserve semantic similarity to text . |
| Outcome: | The proposed attacks are based on two LLM-based attacks that generate natural-looking examples that deceive the target LLM. |
False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalize (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent work has leveraged probing-based approaches to study the separability of malicious and benign inputs in Large Language Models’ internal representations. |
| Approach: | They propose to use probing-based methods to study separability of malicious and benign inputs in LLMs' internal representations to detect harmful and benign content. |
| Outcome: | The proposed methods show that they learn superficial patterns rather than semantic harmfulness. |
VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysis (2026.acl-long)
Copied to clipboard
| Challenge: | Existing metrics for caption evaluation lack factual accuracy and limited context handling . VC-Inspector provides reproducible, fact-aware alternative that aligns closely with human judgments. |
| Approach: | They propose a lightweight, open-source large multimodal model for reference-free evaluation of video captions with a focus on factual accuracy. |
| Outcome: | Experiments show that VC-Inspector can generalize across diverse domains and improve on existing metrics. |